Sparsity Based Regularization

In previous lectures, we saw how regularization can be used to restore the well-posedness of the empirical risk minimization (ERM) problem. We also derived algorithms that use regularization to impose smoothness assumptions on the solution space (as in the case of Tikhonov regularization) or introduce additional structure by confining the solution space to low dimensional manifolds (manifold regularization). In this lecture, we will examine the use of regularization for the achievement of an alternative objective, namely sparsity. During the last ten years, there has been an increased interest in the general field of sparsity. Such interest comes not only from the Machine Learning community, but also from other scientific areas. For example, in Signal Processing sparsity is examined mainly in the context of compressive sensing [CRT06, Dono06] and the so called basis pursuit [CDS96]. In the Statistics literature, basis pursuit is known as the lasso [Tibs96]. Strong connections also exist with sparse coding [OlFi97] and independent component analysis [HyOj00]. In these notes, we discuss sparsity from a regularization point of view, and only refer to these connections as they arise from within this framework. Initially, we motivate the use of sparsity and emphasize on the problem of variable selection. Then, we present the formulation of the sparsity based regularization problem, develop tractable approximations to it, and justify them using a geometric interpretation of sparsity. Finally, after discussion of some of the properties of these approximations, we describe an algorithm for the solution of the sparsity based regularization problem.